Papers with decoding speed
Infusing Sequential Information into Conditional Masked Translation Model with Self-Review Mechanism (2020.coling-main)
Copied to clipboard
| Challenge: | Existing non-autoregressive models generate target words in parallel, but with a large latency due to the left-to-right dependency. |
| Approach: | They propose to train a conditional masked translation model and refine results within several iterations to remedy a flawed translation by non-autoregressive models. |
| Outcome: | The proposed model outperforms state-of-the-art models by over 1 BLEU while using less training computations. |
Subset Retrieval Nearest Neighbor Machine Translation (2023.acl-long)
Copied to clipboard
| Challenge: | k-nearest-neighbor machine translation (kNN-MT) is a new approach to improve NMT performance without additional training. |
| Approach: | They propose a method that integrates example-search into the decoding algorithm to improve neighbor token retrieval. |
| Outcome: | The proposed method achieves a speed-up of up to 132.2 times and an improvement in BLEU score of up 1.6 compared with kNN-MT in the WMT’19 translation task and the domain adaptation tasks in De-En and En-Ja. |
EDITOR: An Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints (2021.tacl-1)
Copied to clipboard
| Challenge: | Empirically, EDITOR uses soft lexical constraints more effectively than the Levenshtein Transformer while speeding up decoding dramatically compared to constrained beam search. |
| Approach: | They propose an Edit-Based TransfOrmer with Repositioning that integrates lexical preferences into output sequences by iterative editing hypotheses. |
| Outcome: | The proposed model uses soft lexical constraints more effectively than the Levenshtein Transformer while speeding up decoding dramatically compared to constrained beam search. |
Director: Generator-Classifiers For Supervised Language Modeling (2022.aacl-main)
Copied to clipboard
| Challenge: | Current language models achieve low perplexity but their resulting generations still suffer from toxic responses, repetitiveness, and contradictions. |
| Approach: | They propose a new language model architecture that uses a language modeling and a classification head for each output token. |
| Outcome: | The proposed model outperforms existing model guiding approaches in terms of accuracy and efficiency. |
Semi-Autoregressive Neural Machine Translation (D18-1)
Copied to clipboard
| Challenge: | Existing approaches to neural machine translation are typically autoregressive but suffer from low parallelizability and thus slow at decoding long sequences. |
| Approach: | They propose a semi-autoregressive Transformer model for fast sequence generation that keeps the autoregressive property in global but relieves in local . |
| Outcome: | The proposed model achieves 5.58 speedup while maintaining 88% translation quality, significantly better than previous non-autoregressive methods. |
A Frustratingly Simple Decoding Method for Neural Text Generation (2024.lrec-main)
Copied to clipboard
| Challenge: | Neural text generation is notorious for repetitive loops and tedious outputs. |
| Approach: | They propose a method that penalizes future generation of repetitive content . they construct an anti-LM based on previously generated text . |
| Outcome: | The proposed method outperforms established baselines in terms of generation quality, decoding speed, and universality. |
Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation (2023.tacl-1)
Copied to clipboard
| Challenge: | Existing non-AutoRegressive (NAR) text generation models lack proper pre-training, making them far behind pre-trained autoregressive models. |
| Approach: | They propose a novel pre-training task to promote prediction consistency in non-autoregressive (NAR) generation. |
| Outcome: | The proposed model outperforms existing pre-trained models and achieves 17 times speedup in throughput. |
Improving Deep Transformer with Depth-Scaled Initialization and Merged Attention (D19-1)
Copied to clipboard
| Challenge: | Existing methods to improve NLP convergence and computational overhead are limited by stacking more layers. |
| Approach: | They propose a depth-scaled initialization method which reduces parameter variance at initialization and reduces output variance of residual connections to ease gradient back-propagation. |
| Outcome: | The proposed method outperforms the base model on translation tasks with five translation directions while matching the decoding speed of the baseline model. |
Chunk-based Nearest Neighbor Machine Translation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Semi-parametric models augment generation with retrieval, but require expensive retrieval operation for every generated token. |
| Approach: | They propose a semi-parametric model which augments generation with retrieval by retrieving tokens from a datastore. |
| Outcome: | The proposed model can retrieve chunks of tokens from the datastore, instead of a single token, with a low decoding speed. |
Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Experimental results show that the Reinforce-NAT system surpasses the baseline NAT system by a significant margin on BLEU without decelerating the decoding speed. |
| Approach: | They propose a sequence-level training method and a Transformer decoder to fuse the target sequential information into the top layer of the decoded Transformer. |
| Outcome: | The proposed model surpasses the baseline NAT system on BLEU without decelerating the decoding speed and achieves comparable translation performance to the autoregressive Transformer model with considerable speedup. |
Lexically Constrained Neural Machine Translation with Levenshtein Transformer (2020.acl-main)
Copied to clipboard
| Challenge: | Existing approaches to incorporate lexical constraints in neural machine translation have been unsuccessful . |
| Approach: | They propose an algorithm that incorporates lexical constraints into neural machine translation. |
| Outcome: | The proposed method improves on English-German datasets without modification . it does not require any modification to the training procedure and can be easily applied at runtime with custom dictionaries. |
CCIM: Cross-modal Cross-lingual Interactive Image Translation (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing research on text image machine translation (TIMT) lacks recognized source language information resulting in a decrease in translation performance. |
| Approach: | They propose a cross-modal cross-lingual interactive model which incorporates source language information by synchronizing source and target language results. |
| Outcome: | The proposed model outperforms end-to-end models and has faster decoding speed with smaller model size than cascade models. |
Can We Achieve High-quality Direct Speech-to-Speech Translation without Parallel Speech Data? (2024.acl-long)
Copied to clipboard
| Challenge: | Existing two-pass direct speech-to-speech translation models require parallel speech data to train, which is challenging to collect. |
| Approach: | They propose a two-pass direct speech-to-speech translation (S2ST) model that decomposes the task into speech- to-text translation (s2TT) and text-tospech (TTS) they propose 'composer' S2ST model that integrates pretrained S2TT and TTS models into a direct S2 ST model. |
| Outcome: | The proposed model integrates pretrained S2TT and TTS models into a direct S2ST model without parallel speech data. |
Template-free Prompt Tuning for Few-shot NER (2022.naacl-main)
Copied to clipboard
| Challenge: | Prompt-based methods have been successfully applied in few-shot learning tasks . however, when applied to token-level labeling tasks, it would be time-consuming to enumerate the template queries over all potential entity spans. |
| Approach: | They propose a method to reformulate NER tasks as LM problems without templates. |
| Outcome: | The proposed method is 30.12 times faster than the template-based method under few-shot settings. |
An Exploration of Arbitrary-Order Sequence Labeling via Energy-Based Inference Networks (2020.emnlp-main)
Copied to clipboard
| Challenge: | Recent work shows that conditional random fields (CRFs) perform well in sequence labeling tasks. |
| Approach: | They propose several high-order energy terms to capture dependencies among labels in sequence labeling . they use convolutional, recurrent, and self-attention networks to construct these energy terms . |
| Outcome: | The proposed approach improves on four sequence labeling tasks while having the same decoding speed as simple classifiers. |
AIN: Fast and Accurate Sequence Labeling with Approximate Inference Network (2020.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to sequence labeling require sequential computation that makes parallelization impossible. |
| Approach: | They propose to employ a parallelizable approximate variational inference algorithm for the CRF model. |
| Outcome: | The proposed approach improves decoding speed and accuracy with long sentences and is parallelizable for faster training and prediction. |
Modeling Context With Linear Attention for Scalable Document-Level Translation (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Document-level machine translation models lack quadratic complexity in the sequence length due to their attention layers. |
| Approach: | They evaluate a recent linear attention model with a sentential gate to promote a recency inductive bias and compare it to open-source document translation. |
| Outcome: | The proposed model significantly improves translation quality on IWSLT 2015 and OpenSubtitles 2018 with similar or better BLEU scores. |
JANUS: Joint Autoregressive and Non-autoregressive Training with Auxiliary Loss for Sequence Generation (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing approaches to train autoregressive and non-autoregressive models only consider relevance of model parameters, ignoring correlations between the two manners. |
| Approach: | They propose a joint autoregressive and non-autoregressive training method using aUxiliary losS to enhance the model performance in both AR and NAR manners simultaneously. |
| Outcome: | The proposed method improves the model performance in both AR and NAR manners and reduces the inference latency. |
Centroid-Based Efficient Minimum Bayes Risk Decoding (2024.findings-acl)
Copied to clipboard
| Challenge: | Minimum Bayes risk (MBR) decoding requires quadratic time since it computes the expected score between a translation hypothesis and all reference translations. |
| Approach: | They propose a centroid-based MBR decoding method that clusters the translations in the feature space and calculates the expected score using the centroids of each cluster. |
| Outcome: | The proposed method outperforms vanilla MBR decoding in translation quality by up to 0.5 COMET in the WMT’22 EnJa, EnDe, EnZh, and WMT'23 Enja translation tasks. |
Treepiece: Faster Semantic Parsing via Tree Tokenization (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Autoregressive (AR) encoder-decoder neural networks are slow in sequential prediction of natural language to machine-readable parse trees. |
| Approach: | They propose a technique that tokenizes a parse tree into subtrees and generates one subtrea per decoding step. |
| Outcome: | The proposed approach shows 4.6 times faster decoding speed and comparable speed but significantly higher accuracy compared to non-autoregressive (NAR) models. |
PEIT: Bridging the Modality Gap with Pre-trained Models for End-to-End Image Translation (2023.acl-long)
Copied to clipboard
| Challenge: | Image translation is a task that translates an image containing text in the source language to the target language. |
| Approach: | They propose an end-to-end image translation framework that bridges the modality gap between visual inputs and textual inputs/outputs of machine translation (MT). |
| Outcome: | The proposed framework outperforms existing models on a large-scale image translation corpus . it significantly outperformed both cascaded and strong models on the e-commerce domain . |
Revisiting Non-Autoregressive Translation at Scale (2023.findings-acl)
Copied to clipboard
| Challenge: | Extensive experiments on two advanced NAT models show scaling can improve translation performance. |
| Approach: | They empirically examine the impact of scaling on NAT behaviors on a large-scale WMT dataset. |
| Outcome: | The proposed model can achieve comparable performance with the scaling model while maintaining the superiority of decoding speed with standard NAT models. |
Question Tells You Where the Answer Is: Intention-aware Long-Context KV Cache Compression (2026.acl-long)
Copied to clipboard
Liang Zhao, Xiaocheng Feng, Weihong Zhong, Lei Huang, Kun Zhu, Baoxin Wang, Dayong Wu, Guoping Hu, Ting Liu, Bing Qin
| Challenge: | Recent methods to reduce the KV cache size fail to identify crucial KVs for generation while excluding others accurately, resulting in severe information loss. |
| Approach: | They propose an intention-aware KV cache eviction method that identifies and retains crucial KVs according to the attention distribution of intention, which semantically reflects the user’s goal and determines which part of the context is relevant. |
| Outcome: | The proposed method can maintain the model performance while reducing the KV cache size from 128K to 2K, leading to a 6.3x increase in decoding speed and 7.8x enhancement in memory efficiency compared to the default setting. |
Amadeus: Autoregressive Model with Bidirectional Attribute Modelling for Symbolic Music (2026.acl-long)
Copied to clipboard
| Challenge: | Existing symbolic music generation models represent musical notes as a sequence of attribute tokens with fixed unidirectional dependencies. |
| Approach: | They propose a symbolic music generation framework that adopts a autoregressive and a discrete diffusion architectures for note attributes. |
| Outcome: | The proposed framework improves state-of-the-art models across objective and subjective metrics. |